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One  promise  of  constructive  simulation  is  in  providing  an  exploratory  environment  within  which  one  can  identify  factors 
critical  to  performance  -  effectiveness  tradeoffs.  We  will  discuss  construction  of  such  a  trade  space  by  focusing  on  the 
methodology  guiding  the  Crew-systems  for  Long-range  Strike  (CSLRS)  program.  The  methodology  uses  an  iterative,  spiral 
process  to  define  the  trade  space,  develop  system  and  operator  descriptions,  parameterize  the  trade  space  and  analyze 
performance  against  requirements.  We  use  the  approach  advocated  by  Rasmussen  (1983;  1985)  to  characterize  the  trade 
space  and  system  requirements,  combined  with  a  constructive  simulation  approach  to  describe  specific  technologies  and 
human  operators,  and  to  analyze  performance  against  requirements.  Four  vectors  through  the  trade  space  have  been 
identified  to  date:  Image  fusion,  synthetic/enhanced  displays,  dynamic  mission  re -planning  and  human-system  integration 
performance  measures.  We  currently  are  using  the  Rasmussen  abstraction  hierarchy,  in  conjunction  with  standard  subject 
matter  expert  interviews,  to  guide  our  definition  of  specific  issues  surrounding  these  vectors.  As  we  identify  specific 
technology  challenges,  models  are  developed  within  the  CART  environment.  The  technology  challenges  are  then  explored 
against  representative  scenarios.  The  new  challenge  in  this  particular  effort  is  to  understand  mission  effectiveness  in  a  2025 
timeframe  and  to  use  this  understanding  to  develop  a  technology  maturation  roadmap  for  the  four  trade  space  vectors 
identified  above.  Constructive  simulations  serve  a  critical  enabling  function  in  this  process  by  allowing  rapid  development 
and  testing  of  alternative  technologies  within  the  context  of  the  specific  trade  space  of  interest.  As  part  of  our  presentation 
we  will  discuss  unique  methodological  requirements  arising  out  of  modeling  and  simulation  activities  targeted  at  future 
technologies  and  scenarios. 


Introduction 

Evaluating  technology  effects  on  advanced  aircraft  systems 
effectiveness  is  a  challenging  problem,  made  more  so  when 
system  effectiveness  is  moderated  by  human  performance 
considerations.  A  promising  way  to  address  this  challenge 
is  through  constructive  simulations.  These  can  provide 
exploratory  environments  that  allow  analysts  to  identify 
factors  critical  to  performance  -  effectiveness  tradeoffs. 
Merely  constructing  simulations,  however,  does  not  solve 
the  technology  evaluation  problem.  Constructive 
simulation,  by  its  nature,  allows  great  flexibility  in 
technology  combination,  level  of  analysis  and  dimensions  of 
evaluation.  These  factors  combine  to  create  a  trade  space 
that  can  be  unmanageably  large.  A  primary  goal  of  our 
methodology  is  to  bound  the  trade-space  in  a  principled 
manner.  Doing  so  allows  technology  evaluations  that  are 
focused  on  the  most  relevant  aspects  of  a  technology 
investment  program.  Our  method  is  based  on  the 
Technology  Identification,  Evaluation  and  Selection  (TIES) 
methodology  for  aircraft  design  articulated  by  Mavris  and 
Kirby  (1999).  TIES  enables  technology  trade-offs  in  the 
early  stages  of  aircraft  design  by  relying  on  accepted 
engineering  models  of,  for  example,  propulsion,  materials 
and  aeronautics.  Reliance  on  such  models  allows  an 
engineering  design  team  to  conduct  the  physical  modeling 
and  design  of  experiment  activities  required  to  assess  the 
role  of  technologies  on  predicted  aircraft  performance 
through  their  effects  on  structures,  wing  sizes  and  loadings, 
and  propulsion  as  well  as  on  well-behaved  physical 
parameters  like  lift  and  drag.  While  the  TIES  methodology 
seems  well-suited  to  technology  prediction  in  traditional 
engineering  domains,  it  exhibits  shortcomings  when  applied 


to  crew-system  integration  applications.  First,  few 
engineering  models  exist  for  human  performance  that  lend 
themselves  to  predictions  of  technology  effects.  There  are 
several  human  cognitive  performance  architectures  currently 
used  to  model  behavior  in  domains  similar  to  military 
aircraft  operations.  These  include  ACT-R  (Anderson  and 
Lebiere,  1998),  SOAR  (Rosenbloom,  2000),  SAMPLE 
(Zacharias,  2000)  among  others.  However,  some  of  these 
architectures  rely  on  human  performance  data  that  is 
somewhat  controversial  or  require  modeling  at  such  a  low 
level  that  their  use  for  early-stage  technology  evaluations  is 
limited.  Others  manage  these  problems  by  either  “turning 
off’  behavioral  functions  or  limiting  their  levels  of  analysis 
to  broad  behavioral  aggregates  that  makes  informative 
technology  evaluation  difficult.  Second,  critical  portions  of 
the  TIES  methodology  rely  on  expert  judgment.  For 
example,  development  of  a  technology  impact  matrix  (to  be 
discussed  in  detail  later)  is  based  on  consultation  with 
subject  matter  experts  in  sub-disciplines  of  aircraft 
configuration  as  well  as  in  the  technologies  under 
evaluation.  These  experts  rely  on  both  their  own  analysis 
and  on  disciplinary  models  and  historical  data  in  making 
predictions  of  performance  changes  correlated  with  each 
technology.  Because  these  models  are  deterministic  the 
predictions  are  more  reliable  than  would  be  the  case  if  the 
models  were  stochastic,  as  is  the  case  in  the  human  behavior 
representation  community. 

Furthermore,  few  of  the  parameters  used  in  accounting  for 
human  performance  are  sufficiently  well-behaved  to  support 
performance  predictions  in  the  presence  of  new 
technologies.  This  weakens  the  TIES  methodology  as 
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presented  by  Mavris  and  Kirby  by  preventing  use  of  both 
accepted  models  and  the  design  of  experiments  method. 
Simply  put,  the  problem  space  for  human  performance  is 
intractably  large  for  a  traditional  TIES  approach. 

We  attempted  to  address  these  shortcomings  by  integrating 
the  TIES  methodology  with  CART’s  task  network  and 
human  performance  modeling  capabilities.  Our 
methodology  relies  on  a  series  of  principled  analytical  steps 
to  define  a  subset  of  the  possible  trade  space  that  represents 
the  most  informative  technology  evaluation  possible.  Each 
step  in  the  methodology  is  designed  to  constrain  one 
dimension  of  the  trade  space.  Once  identified,  these  are 
then  combined  into  an  evaluation  environment  and 
represented  within  CART.  Technology  alternatives  are  then 
compared  against  a  baseline  and  each  other  to  assess  overall 
impact  on  crew-system  effectiveness. 

Our  methodology  is  shown  in  Figure  1.  In  this  section  we 
briefly  describe  the  methodology  at  a  high  level. 
Subsequent  sections  will  discuss  each  step  in  greater  detail. 


Figure  1.  Methodology  Overview 

We  begin  with  development  of  a  general  scenario  and  a  set 
of  excursions  capturing  plausible  operational  variations. 
The  scenario  provides  a  structure  within  which  the 
technology  evaluation  is  situated.  The  absence  of  a  scenario 
would  weaken  the  validity  of  predictions  regarding  the 
performance  of  the  crew.  The  need  to  situate  our 
evaluations  of  new  technologies  leads  to  step  2, 
development  of  platform  alternatives.  This  is  one  way  in 
which  our  methodology  differs  from  that  of  Mavris  and 
Kirby  (1999).  Our  technology  evaluations  required 
developing  alternative  platform  concepts  because  of 
significant  interactions  between  the  platform  and  specific 
missions.  For  example,  platform  concepts  involving  (1)  a 
low-altitude,  subsonic  aircraft  versus  (2)  a  high-altitude 
hypersonic  aircraft  are  likely  to  perform  very  differently 
across  several  types  of  long-range  strike  missions.  All  of 
these  interactions  must  be  considered  in  an  overall 
evaluation.  Our  next  step  is  to  develop  a  morphological 


matrix.  This  provides  a  structured  method  of  identifying 
technology  combinations  from  among  many  potentially 
useful  candidates  (Twiss,  1992).  The  universe  of  possible 
technologies  is  constrained  by  including  only  those  that 
relate  to  the  fundamental  characteristics  of  the  system  under 
evaluation.  Since  the  fundamental  characteristics  of  a 
propulsion  system  differ  from  those  of  a  crew-system 
technology  alternatives  appearing  in  respective 
morphological  matrices  also  will  differ.  When  candidate 
technology  alternatives  have  been  identified,  they  are 
aggregated  into  a  set  of  platform  alternatives.  These  are 
combined  with  the  mission  excursions  into  a  CSI  challenge 
matrix.  Challenges  specific  to  platform-excursion 
combinations  are  then  identified.  Summary  challenges 
across  this  space  are  then  identified  using  a  set  of  rules  to  be 
discussed  below.  Technology  dimensions  and  disciplinary 
metrics  are  identified  for  each  of  the  CSI  challenges  in  the 
next  step.  This  information  is  combined  with  the  platform 
alternatives.  CART  models  are  then  developed  to  support 
evaluations  of  platform  effects  on  mission  performance. 
The  final  step  is  to  conduct  gap  analyses  comparing  each 
platform  to  baseline  technology  and  to  each  other.  The 
remainder  of  the  paper  discusses  each  step  of  the 
methodology  in  detail. 

Methodology  Application  and  Results 
Scenario/Excursion  Construction.  Our  method  begins  with 
construction  of  a  base  scenario  and  excursions  from  this 
base.  Scenario  construction  serves  as  the  fundamental 
problem  definition  activity  within  which  technology 
evaluation  is  situated.  This  establishes  the  context  and 
boundaries  of  subsequent  performance  and  technology 
evaluations.  Our  scenario  was  constructed  by  envisioning  a 
standard  geo-political  background  leading  to  a  plausible 
military  confrontation.  The  background  context  was 
populated  with  technologies  that  an  adversary  can  be 
expected  to  possess  within  a  timeframe  approximately  15 
years  in  the  future.  This  information  was  obtained  through 
open  sources.  These  technologies  were  “placed”  within  the 
scenario  to  present  challenges  to  the  platforms  that  would  be 
evaluated  as  part  of  our  technology  selection.  We  then 
developed  a  baseline  mission  and  associated  flight  profile. 
Excursions  then  were  defined  to  investigate  performance 
under  several  standard  mission  variations  within  current  Air 
Force  doctrine.  Five  excursions  were  included:  Flarbor 
mining,  runway  suppression,  threat  suppression,  time- 
critical  targeting  and  attack  of  a  hardened  tunnel  containing 
intermediate  range  missiles. 

Morphological  Matrix  for  Technology’  Concept 
Identification.  Mavris  and  Kirby  (1999)  define  a 
morphological  matrix  as  a  structured  means  of  decomposing 
a  system  into  combinations  of  conceptual  subsystems  that 
will  satisfy  mission  requirements.  For  this  decomposition  to 
be  productive,  it  is  necessary  to  identify  a  critical  set  of 
characteristics  that  will  be  addressed  by  the  technology 
combinations.  In  our  case  this  set  is  encompassed  by  the 
crew  and  crew-system.  We  identified  6  characteristics  of  a 
long-range  strike  crew  system:  flight  control,  command  and 
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control,  situation  awareness,  location/orientation,  safety  and 
lethality.  We  then  combined  these  with  alternative 
performance  attributes  into  a  morphological  matrix.  The 
matrix  is  shown  in  Figure  2.  Cells  of  the  matrix  were 
populated  with  alternative  attributes  of  each  characteristic. 
For  example,  flight  control  alternatives  include  automatic 
control,  manual  control  and  hybrid  means  of  control. 
Combining  attributes  from  the  resulting  table  yields 
individual  concepts  for  further  study.  Although  it  is 
possible  to  generate  a  large  number  of  concepts  from  the 
morphological  matrix  the  number  of  plausible  concepts 
typically  will  be  smaller  than  the  factorial  combination  of  all 
tabled  attributes.  Three  platform  alternatives  were  chosen 
for  this  evaluation:  arsenal  platform,  low  supersonic  and 
high  supersonic.  These  alternatives  represent  a  wide  range 
of  capabilities  both  with  respect  to  fundamental  dimensions 
of  an  airframe:  Altitude,  airspeed,  weapon  mix  and  capacity, 
threat  management  strategies,  communication  capabilities, 
and  with  respect  to  the  crew-system  characteristics  shown 
on  the  left  of  matrix. 
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Figure  2.  Morphological  Matrix 


Identify  Crew-system  Integration  (CSI)  Challenges. 
Construction  of  a  morphological  matrix  allows  analysts  to 
define  a  concept  space.  Flowever,  further  progress  will  not 
be  possible  unless  specific  CSI  challenges  are  identified.  It 
is  these  CSI  challenges  that  the  technologies  are  evaluated 
against.  Accordingly,  the  next  step  involved  construction  of 
a  platform  by  mission  excursion  table  populated  with  CSI 
challenges.  Figure  3  provides  a  partial  example  of  such  a 
table.  The  content  of  this  table  is  developed  by  means  of  an 
analytical  process  involving  experts  in  mission 
requirements,  platform  characteristics  and  crew-system 
integration.  The  process  was  guided  by  consideration  of  the 
LRS  crew-system  characteristics  identified  during 
morphological  matrix  definition.  For  example,  an  arsenal 
platform  executing  a  runway  suppression  mission  exhibits 
specific  challenges  in  the  areas  of  situational  awareness, 
safety  against  threats  and  target  lethality.  These  were 
placed  into  the  appropriate  cells  of  the  table  shown  in 
Figure  3.  It  should  be  noted  at  this  point  that  not  all  cells  in 


a  CSI  challenge  matrix  will  necessarily  be  completed.  It  is 
possible,  in  fact  probable,  that  some  platform  concepts  will 
be  inappropriate  for  particular  missions.  Thus,  the  arsenal 
platform  is  unlikely  to  be  appropriate  for  a  harbor  mining 
mission. 

Note  that  Figure  3  also  contains  marginal  cells  representing 
summary  information  across  both  mission  types  and 
platforms.  Summarizing  across  missions  and  across 
platforms  allowed  us  to  identify  manageable  subsets  of  CSI 
challenges  affecting  multiple  platforms  and  missions. 
Further  summarizing  across  the  marginals  produced  a 
canonical  set  of  challenges  that  supported  the  task  network 
modeling  to  be  discussed  below.  Creation  of  the  summary 
platform/mission  challenges  or  the  canonical  challenge  set  is 
not  simply  a  matter  of  enumerating  the  contents  of  cells  in 
the  table.  Rather,  we  discovered  several  rules  that  were 
useful  in  guiding  the  process.  First,  challenges  that  appear 
consistently  across  cells  should  be  captured.  Second, 
technology  availability  should  be  considered.  CSI 
challenges  that  seem  to  be  satisfied  by  currently  available 
(old)  technology  should  be  eliminated  from  further 
consideration.  Third,  mission  phase  criticality  affected  the 
“weighting”  of  some  CSI  challenges.  Since  some  mission 
phases  are  arguably  more  important  than  others  (e.g.,  threat 
management  over  cruise)  the  challenges  associated  with  the 
former  phases  should  be  preferred  in  the  summarization 
process.  Fourth,  challenges  that  are  related  to  brittle 
technologies  should  appear  in  summaries.  An  example  of  a 
brittle  technology  would  be  that  of  communication. 
Although  current  communication  capabilities  are 
impressive,  they  also  can  be  easily  disrupted.  Disruption 
would  lead  to  potential  mission  degradations  that  should  be 
explored  with  new  technology  concepts.  Fifth,  the 
perceived  cost  of  new  technologies  should  be  considered. 
While  it  is  possible,  for  example,  to  imagine  particle  beam 
weapons  as  potential  solutions  to  some  challenges  the  cost 
of  such  technologies  is  unlikely  to  make  them  viable  within 
the  time  frame  under  consideration.  It  probably  would  not 
be  productive  to  include  CSI  challenges  related  to  these 
technologies  in  subsequent  analysis.  Sixth,  challenges  that 
might  be  associated  with  technologies  having  particularly 
high  payoff  were  added  to  the  summaries.  Based  on  the 
mission  execution  challenges  identified  in  Figure  3,  in 
conjunction  with  the  summarization  rules  discussed  above, 
four  crew-system  integration  (CSI)  areas  were  identified  as 
critical  to  mission  success.  These  were  image  fusion, 
synthetic/enhanced  vision,  dynamic  mission  re-planning  and 
human-system  technology  integration. 

System  Effectiveness  Metric  Identification.  Development  of 
the  morphological  matrix  results  in  identification  of 
aggregate  technology  concepts.  The  CSI  challenge  matrix 
helps  identify  CSI  areas  critical  to  mission  success.  Our 
next  step  was  to  relate  this  technology  information  to  human 


3 


Mission  Excursion 

Arsenal 

Low  supersonic 

High  supersonic 

Mission  Challenge 
Summary 

Harbor  mining 

N/A 

•  Pop-up  threat 
detection 

•  Mine  placement  to 
planned  coordinates 

N/A 

•  Weapon  delivery 
accuracy 

•  Threat  management 

Runway  suppression 

•  Weapon  guidance 

•  BDA 

•  Coordinate  updates 

•  Effective  target 
coordinate  updates 

•  Effective  SAR 
imaging 

•  Weapon  delivery  to 
precise  coordinates 
from  mach  10 

•  Weapon  guidance 

•  Effective  BDA 

•  Multiple  image 
sources 

•  Mission  updating 

SEAD 

•  Weapon  guidance 
over  the  horizon  must 
be  highly  accurate 

•  Multiple  targets  with 
coordinated  attack 
management 

•  Protect  against 
passive  detection 

•  Effectively  merge 
SAR,  EO,  tactical 
and  human  visual 
information 

•  Target  coordinate 
updates  will  be 
crucial 

•  Target  imagery 

•  Target  location 
updates 

•  Multiple  image 
sources 

•  Weapon  guidance 
and  control 

Mobile  TCT 

N/A 

•  Maintain  SA  in  real¬ 
time  environment 

•  Effective  integration 
with  all  ISR  assets 

•  Target  updates  must 
be  timely 

•  ISR  integration 

Hardened  tunnels 

•  Weapon  guidance 
over  the  horizon  must 
be  highly  accurate 

•  BDA 

•  Must  have  accurate 
target  coordinates  at 
point  of  release 

•  BDA 

•  Weapon  guidance  on¬ 
board? 

•  Target  coordinate 
accuracies 

•  BDA 

Platform  Challenge 
Summary 

•  Over  the  horizon 

BDA 

•  Long  range  weapon 
guidance 

•  Target/coordinate 
updates 

•  Pop-up  threats 

•  Imaging  effectiveness 

•  ISR  integration 

•  High  speed  weapon 
delivery 

•  Mission  update 
timeliness 

•  Dynamic  re-planning 

•  Image  fusion 

•  Synthetic/enhanced 

vision 

•  Human-system 

integration 

Figure  3.  CSI  Challenges 


performance  by  constructing  a  technology  impact  matrix 
(Mavris  and  Kirby,  1999),  an  example  of  which  is  shown  in 
Figure  4.  This  was  done  in  a  2-step  process.  First, 
technology  dimensions  of  each  CSI  challenge  were 
identified.  This  process  relied  primarily  on  expert 
judgments  and  literature  review.  For  example,  the  critical 
dimensions  of  dynamic  mission  re-planning  include  network 
architecture,  uplink/downlink  method,  data  throughput  rate, 
data  structure,  and  data  sources.  Second,  disciplinary 
metrics  were  identified  for  each  technology  dimension. 
These  metrics  are  referred  to  as  k-factors  by  Mavris  and 
Kirby  and  represent  the  critical  addition  at  this  point  in 
evaluation.  Figure  4  displays  the  technology  impact  matrix 
for  dynamic  mission  re -planning.  Three  disciplinary 
metrics  are  shown  for  the  technology  dimension  of 
uplink/downlink  method:  (1)  time  needed  to  input  new 
information  into  on-board  automated  systems  (update  time), 
(2)  time  input  accuracy  during  mission  updating,  and  (3)  the 
ratio  of  manual  to  automated  updating  required  (update 
efficiency).  In  general,  as  well  as  for  the  evaluations 
discussed  here,  sources  of  the  k-factor  vector  for  areas  under 
evaluation  include  interviews  with  subject-matter  experts, 
literature  reviews,  the  expertise  of  analysts  involved  in  the 


evaluations  and  so  on.  Note  that  the  combination  of 
technology  dimensions  of  the  CSI  challenges  and 
technology  concepts  aggregated  into  platforms  constrains 
the  technologies  available  for  consideration  during  this 
phase  of  overall  evaluation.  Maintaining  this  constraint  is 
important  in  preventing  a  run-away  consideration  of  all 
possible  technologies,  though  one  can  always  go  back  to  the 
morphological  matrix  to  formulate  new  concepts  if  desired. 
Two  other  factors  help  to  maintain  this  constraint:  (1) 
consideration  of  technologies  rather  than  specific 
applications  of  technologies  and  (2)  keeping  the  evaluations 
situated  strictly  within  the  scenario  excursions  defined  in  the 
first  phase. 
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Figure  4.  Technology  Impact  Matrix 


CART  Modeling.  Figure  4  sets  the  stage  for  the  technology 
evaluations.  However,  in  many  instances,  including  those 
discussed  by  Mavris  and  Kirby  (1999),  these  evaluations 
rely  on  either  well-documented  engineering  models  or  on 
the  predictions  of  experts  in  the  effects  of  the  technologies 
being  evaluated.  In  the  case  of  human-systems  integration, 
however,  predictions  based  on  these  sources  either  are 
unavailable  or  of  suspect  reliability.  The  CART 
environment,  as  described  by  Martin,  Brett  and  Hoagland 
(1999)  provides  a  way  to  empirically  conduct  the 
evaluations  needed  to  make  technology  investment 
recommendations. 

The  CART  environment  is  a  goal-oriented,  task  network 
modeling  tool  based  on  the  IMPRINT  discrete  event 
simulator.  It  allows  analysts  to  develop  hierarchical  task 
networks  describing  the  structure  of  mission  execution  to  an 
arbitrary  level  of  detail.  The  task  network  approach 
provided  by  IMPRINT,  combined  with  an  ability  to 
integrate  tasks  with  goal  descriptions,  allows  one  to  evaluate 
a  range  of  technologies  by  describing  their  effects  on 
operator  performance  at  whatever  level  of  analysis  is 
desired.  Within  the  context  of  the  CART  environment 
technology  effects  are  apparent  in  two  ways. 

First,  by  affecting  task  network  structure  a  particular 
technology  can  affect  mission  outcomes  by  adding,  deleting 
or  changing  the  ordering  of  network  elements.  For  example, 
a  mission  updating  technology  that  automatically  inputs 
target  coordinates  into  a  flight  computer  might  result  in  the 
elimination  of  several  tasks  on  the  part  of  an  aircrew.  The 
resulting  effect  on  a  running  CART  model  might  be  an 
increase  in  target  coordinate  recording  accuracy,  a  decrease 
in  time  required  to  register  new  coordinates  with  the  flight 
computer,  an  increase  in  the  number  of  targets  that  can  be 
entered  into  the  flight  computer  per  unit  time  and  so  on.  At 
the  same  time,  automating  mission  updates  might  require  an 
addition  of  new  tasks  that  might  also  affect  accuracy,  time 
and  efficiency.  These  tradeoffs,  and  the  resulting  effects  on 


mission  effectiveness,  are  explored  by  modeling  the  task 
network  associated  with  the  technology  in  CART. 

Second,  a  technology  can  affect  the  performance  values 
associated  with  task  execution.  This  can  happen  when  a 
technology  influences  the  accuracy  with  which  a  task  can  be 
carried  out  or  the  reliability  with  which  the  task  is 
performed.  An  example  of  an  accuracy  effect  was  found 
with  technologies  requiring  manual  re-programming  of 
target  coordinates.  As  the  number  of  coordinates  requiring 
updating  increased,  the  accuracy  of  updates  decreased  due 
to  working  memory  load.  An  example  of  how  technology 
affects  task  execution  reliability  was  observed  with 
technologies  that  relied  on  satellites  for  communication  of 
mission  updates.  When  communication  mediated  by  these 
satellites  was  disrupted,  other  methods  were  used. 
However,  task  execution  performance  became  much  more 
variable  with  the  result  that  overall  mission  performance 
degraded  across  multiple  mission  simulations.  CART 
allows  an  analyst  to  vary  these  two  parameters  by  (1) 
defining  mean  accuracy  associated  with  a  particular  task  and 
(2)  specifying  the  distribution  and  variance  associated  with 
a  task. 

Figure  4  summarizes  the  manner  in  which  model-based 
evaluations  were  conducted  for  dynamic  mission  re¬ 
planning.  A  disciplinary  metric  vector  was  defined  for  each 
of  the  technology  dimensions  identified  in  earlier  phases.  A 
subset  of  the  dimensions  is  shown  in  the  Technology  Impact 
Matrix.  For  example,  metrics  for  the  uplink/downlink 
method  included  the  time  required  to  complete  updates,  the 
input  accuracy  associated  with  information  entry,  and 
update  efficiency:  A  measure  of  how  many  updates  could  be 
successfully  executed  per  unit  time.  CART  models  were 
developed  for  each  of  the  three  platforms  representing 
technologies  to  be  evaluated.  Model  runs  then  were 
conducted.  The  resulting  effects  on  each  of  the  disciplinary 
metrics  would  be  shown  in  Figure  4  as  either  positive  or 
negative  percentage  changes  in  each  metric,  relative  to 
performance  in  a  baseline  system.  This  information  would 
then  support  the  gap  analysis  discussed  below. 

Comparative  Gap  Analysis.  The  final  step  of  our 
methodology  is  to  assess  the  mission  effectiveness 
associated  with  the  technologies  comprising  our  3 
evaluation  platforms.  To  do  this  we  develop  a  model  of 
each  system  metric  based  on  the  vector  elements  of 
Figure  5.  In  most  cases  we  anticipate  that  these  models  will 
take  the  following  form: 

R  =  b0  +  Xbik;  +  Xbi.kf  +  ZZb.jk.kj 

In  this  form  R  represents  a  system  metric,  bj  represents 
linear  regression  coefficients,  bj;  represents  quadratic 
coefficients,  by  represent  cross-product  coefficients  and  each 
k  term  represents  k-factors  from  the  parameter  vector  in 
Figure  4. 
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Discussion 

The  methodology  discussed  above  provides  a  principled 
way  of  moving  from  an  open-ended  space  of  potential 
technology  concepts  to  predictions  of  technology  effects  on 
human-system  effectiveness.  The  value  of  this  method 
resides  in  providing:  (1)  a  way  to  define  a  bounded  trade- 
space  within  which  technology  alternatives  can  be 
identified,  (2)  a  method  of  combining  mission  requirements 
with  high-level  trade-space  technology  alternatives  to  assist 
in  identifying  areas  to  help  focus  technology  predictions,  (3) 
a  modeling  tool  to  represent  the  task  networks  required  to 
carry  out  mission  requirements  using  identified  technology 
concepts  and  (4)  an  analysis  method  relating  technologies 
directly  to  crew-system  effectiveness  metrics.  Future  work 
will  be  concentrated  in  several  areas.  The  first  addresses 
how  to  represent  the  technology  under  review.  If 
representing  technology  can  properly  be  considered  a 
hierarchy,  then  the  question  becomes  what  level  of  analysis 
is  meaningful.  Our  technology  representations  to  date  have 
primarily  been  functional  and  have  occupied  a  fairly  high 
level  of  aggregation.  However,  it  is  possible  to  represent 
technology  on  at  least  two  lower  levels:  Technology  class 
level  and  a  specific  system  level.  These  levels,  when 
applied  to  radar  for  example,  might  include  sensing,  SAR 
and  a  specific  system  or  application.  Choosing  one 
representational  level  or  another  when  executing  the 
methodology  outlined  here  would  lead  to  different 
simulation  outcomes. 

A  similar  question  arises  in  consideration  of  human 
performance.  Again,  our  simulations  to  date  have  been 
limited  to  relatively  high  levels  of  analysis  in  our  human 
performance  modeling.  While  maintaining  this  level  has 
been  useful  in  comparing  platform  configurations  and  in 
identifying  broad  crew-system  effectiveness  issues,  it  is  less 
useful  in  evaluating  specific  alternatives  (e.g.,  SAR  versus 
EO  as  a  target  recognition  aid).  On  the  other  hand,  moving 
the  human  performance  level  of  analysis  to  a  low  level  risks 
(1)  creating  a  problem  of  proliferating  process  boxes  and  (2) 
disconnecting  human  performance  evaluations  from  an 
overall  understanding  of  system  effectiveness.  Worse  still, 
the  levels  of  analysis  problems  for  technology  and  human 
performance  are  interrelated. 

A  third  challenge  for  future  work  with  this  methodology  is 
how  to  account  for  variation  in  crew  performance,  at  both 
the  task  and  method  levels.  Crews  often  achieve  goals  by 
combining  tasks  in  different  sequences  or  by  using  different 
tasks  altogether.  Additionally,  tasks  can  be  accomplished 
through  variations  in  methods.  This  variability  creates 
problems  for  technology  evaluation,  as  it  is  difficult  to  state 
with  certainty  that  one  combination  of  technology  is  clearly 
superior  to  another  without  regard  to  task  and  method 
variation. 


Finally,  the  problem  of  technology  interactions  must  be 
addressed.  As  has  been  pointed  out  by  Overdorf  (2002), 
technologies  might  lead  to  performance  improvements  when 
considered  in  isolation  but  to  performance  degradations 
when  combined.  This  raises  the  problem  of  combinatorial 
explosion  in  which  all  possible  combinations  of  all 
technology  candidates  must  be  considered.  We  finessed  this 
problem  in  the  current  study  by  combining  technologies  into 
functionally  defined  platforms  defined  to  address  only  the 
mission  excursions  of  our  scenario.  However,  this  strategy 
is  more  ad  hoc  than  we  would  like. 

We  feel  that  the  methodology  discussed  here,  particularly 
the  addition  of  the  CART  modeling  and  simulation 
environment,  holds  great  promise  for  evaluating  technology 
effects  on  crew-system  performance.  As  the  challenges 
outlined  above  are  addressed,  the  general  method  should 
greatly  facilitate  technology  investment  decisions  and  the 
place  of  crew-systems  in  simulation-based  acquisition. 
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